CONWAY&McCLAIN The Conduct of an Effective Simulation Study The Conduct of an Effective Simulation Study

Although the software tools to develop simulation models have improved dramatically, relatively little has been written about how to most effectively exercise those models. This paper provides warnings and suggestions to improve simulation studies. Although based primarily on the authors’ experience with discrete-event models in manufacturing situations, many of their observations are more widely applicable. Editor’s note: This is a pdf copy of an html document which resides at http://ite.pubs.informs.org/Vol3No3/ConwayMcClain/ Educators are facing a new challenge increasing demand for including simulation in curricula, as complete courses or as a topic in existing courses. Simulation is no longer for specialists only. Graphical simulation packages such as Extend and Arena, and spreadsheet-based versions such as Crystal Ball and @Risk have put this tool into the hands of end-users, and into many more courses in business and engineering. (For a review, see Swain, 2001.) Applications abound. A quick visit to the web sites of some of the software suppliers reveals some of the volume and variety of situations where simulation has been used, including Six-sigma training, real options analysis, financial planning, probabilistic oil and gas reserve estimates, cost estimation for the Health Insurance Portability and Accountability Act, process operating costs in mining, and credit risk of corporate restructuring. The authors have used simulation in manufacturing and health systems, both in application and in research. These packages make it possible to construct models of reasonably complex systems with relative ease. No longer do the time, effort and expertise required make the technique economically viable only for major projects. In education, students can begin modeling almost immediately, with only minimal instruction about the package they are using. In industry, the person with the problem can conduct the study without having to engage the services of a professional. However, this ease of use can be deceptive it is now easier than ever to produce sophisticatedappearing results that are entirely misleading. We saw one example in which the simulation was used to estimate how long customers would have to wait. The model did a good job of representing the real situation, but the conduct of the study led to incorrect results. Specifically, when the rate of customer arrivals was increased to investigate a worst-case scenario (a very good idea) the average waiting time did not increase substantially (wrong). The problem was that the run-length (how long the simulation was observed) was too short to see what would eventually happen after a large increase in the customer load. A longer run-length showed that the customer wait was enormous in the worst-case scenario, and in several other scenarios as well. If the first results had been used to design the actual service system, the outcome would have been a customer-satisfaction disaster. Effectively constructing a simulation model, and using it to learn something interesting and useful about a real system, is a challenging task. It requires a combination of competence, ingenuity and experience that is neither intuitively obvious nor trivially learned. Using this tool with inadequate training incurs a risk not unlike that of placing a sharp tool in the hands of inexperienced users. There is some prospect of accomplishing useful work, but also danger of doing harm. The title of the first simulation text (Tocher, 1963) is The art of simulation, which nicely reflects one of our main points simulation packages are tools that require care and skill in their application. Fortunately, enduser simulation programs can facilitate teaching and learning because of the relative ease in constructing, running and changing models, and the ready access to both animation and statistical summaries to see what is actually happening. However, the danger remains of producing sophisticated-looking results that are misleading INFORMS Transaction on Education 3:3 (13-22) 13 c ©INFORMS ISSN: 1532-0545